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Abstract 

The learning management system (LMS) has become a critical tool for nearly all institutions of 
higher education, and a driving force in online learning. According to a 2014 report by the 
Educause Center for Analysis and Research, 99% of higher education institutions have an LMS in 
place, and the LMS is used by 85% of faculty and 83% of students. This was not always the case, 
however. There was a time in the not-so-distant past when using an LMS was considered highly 
innovative. Understanding the growth and adoption of the LMS is a stepping stone to 
understanding how faculty may choose to adopt other technological and pedagogical innovations. 
This study was conducted at a large, research-intensive public university in the Midwest, which 
has used the same LMS for 15 years. Prom a small pilot, adoption has grown to nearly universal 
use. This study used system logs and database queries to examine how faculty used the LMS. The 
results identified the features that were used most frequently and how usage had changed over 
time. In addition, the study compared the usage data for face-to-face and online courses to 
determine if there are differences in LMS use due to course modality. Based on this, it is possible 
to better understand the role the LMS plays in higher education and online learning, to inform 
development of next generation learning systems or other innovative technologies. 
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Introduction 

Educators have long sought to infuse contemporary innovations into teaching and learning. 
Through the years, various print, audio, video, and computer technologies have been incorporated 
into education and training (Lever-Duffy, McDonald, & Mizell, 2003; Picciano, 2001). With each 
wave of technological innovation, tools become more seamless and instructional practices evolve. 
As the Internet came of age, Learning Management System (LMS) technology became widely 
available, enabling faculty with little technical skill to deliver instruction to students at a distance. 
Most higher education institutions have since integrated the LMS with other institutional 
infrastructure systems, encouraged faculty adoption of the LMS, and provided the necessary user 
training and support. Understanding the growth and adoption of the LMS is a stepping stone to 
understanding how faculty may choose to adopt other technological and pedagogical innovations. 
This study at a large, research-intensive public university in the Midwest, which has used the same 
LMS for 15 years, examined which features were used most frequently and how usage has changed 
over time. 
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History of Learning Management Systems 

The Learning Management System (LMS), also referred to as Course Management System 
(CMS) or Virtual Learning Environment (VLE), has evolved over decades of technological 
innovation to become a cornerstone of institutional instructional technology infrastructure. With 
roots dating back to the first computer-assisted instruction system, PLATO, developed in the 
1960’s (Bitzer, Braunfeld, & Lichtenberger, 1961), the LMS was popularized with the advent of 
the Internet, with many LMS platforms available today (Hill, 2017). LMS platforms commonly 
include a suite of integrated tools that enable online delivery of instructional content, interaction 
and collaboration, and tracking and reporting of student participation (Rouse, 2005). The LMS has 
been touted as having been instrumental in extending instruction and access to learning beyond 
the physical classroom (Harrington, Staffo, & Wright, 2006), enabling secure online collaborations 
among faculty and students (Barron, 2003; Liu, 2005), and promoting the integration of otherwise 
discrete content delivery, course management, and student assessment functions (Linder, Bruenjes, 
& Smith, 2017; Weaver, Spratt, & Nair, 2008). 

The longstanding and steady growth of LMS use in higher education has been well 
documented (Cannean & Haefner, 2003; Daniels, 2009; Harrington et al., 2006; Malikowski, 
2010; Mkhize, Mtsweni, & Buthelezi, 2016; Mills, 2005; Vovides, Y., Sanchez-Alonso, S., 
Mitropoulou, V., & Nickmans, G.; West, Waddoups, & Graham, 2007). With learning 
management systems being present at 99% of higher education institutions, their use is now 
ubiquitous (Dahlstrom, Brooks, & Bischel, 2014; Green, 2013; Lang & Pirani, 2014). Recent 
studies of higher education students and their technology preferences have noted that nearly all 
students use an LMS and that the LMS is consistently identified by students as among the most 
important instructional technologies for their academic success (Brooks, 2016; Dahlstrom, 
Walker, & Dziuban, 2013). 

The LMS remains a mainstay of online education infrastructure, with 85 percent of faculty 
confirming their use of the institution's LMS (Brooks, 2015) and 81 percent of chief online 
education officers reporting the LMS to be the technology that is most important to online 
programs (Legon & Garrett, 2017). Institutional leaders recognize the importance of supporting 
faculty in their use of instructional technologies including the LMS, with faculty development 
ranked as the number one key issue in teaching and learning in 2017 (Educause Learning Initiative, 
2017). 

Despite its widespread adoption, some have questioned whether the LMS is still needed. 
Education futurists call for LMS tools and platfonns to be more agile to support emerging 
instructional practices, as some wish to unbundle the components of a learning experience to remix 
open content and educational apps in new ways (Adams Becker et al., 2017; Anshari et al., 2016; 
Garcia-Penalvo & Fonnent, 2014). Some thought leaders feel current learning management 
systems are too limited in functionality (Brown, Dehoney, & Millichap, 2015a) and have proposed 
a next-generation LMS, often referred to as a "next-generation digital learning environment" 
(NGDLE), to support more personalized and flexible learning experiences (Brown, 2015). Rather 
than being a single system, the NGDLE would encompass a "confederation of IT systems and 
application components that adhere to common standards...that would enable diversity while 
fostering coherence" (Educause Learning Initiative, 2015). 

With the maturation of the LMS and critical mass adoption across higher education, there 
is a need to look deeper into how the LMS is being used, to see if it is still meeting the needs of 
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faculty and students. Measurement of LMS usage, however, is challenging and often relies on 
estimates based on guesswork. This study addresses this problem by analyzing faculty usage of 
the LMS via empirical data gathered directly from the LMS database, and without the adoption of 
sophisticated yet expensive analytics systems. This provides a means to follow the digital 
footprints faculty leave in the LMS as a reflection of their course design. 


Review of Related Literature 

Previous studies have sought to understand the motivations and experiences of instructors 
as they adopt a learning management system (Samarawickrema & Stacey, 2007; West, Waddoups, 
& Graham, 2007) and their overall satisfaction with the tools available (Walker, Lindner, 
Murphrey, & Dooley, 2016; West, Waddoups, Kennedy, & Graham, 2007), focusing largely on 
the attitudes and observations shared by faculty themselves. Not surprisingly, faculty experiences 
have often varied widely from one institution to another, as enabling and impeding factors differ. 
Even so, research on the patterns of faculty adoption of the LMS and the degree to which the 
available capabilities meet instructional needs have been valuable. The data from such studies have 
provided interesting insights, but have inherent limitations because they often relied on instructor 
perceptions through self-reported usage. 

Malikowski (2008) took an alternative tactic in seeking to understand the factors related to 
breadth of LMS feature use, manually tallying tool usage frequency in LMS courses from across 
842 course websites volunteered for study by 394 faculty members over a three-year period. He 
selected six features of the LMS and recorded the number of times each was utilized within the 
sample of courses. Through his descriptive and inferential statistical analysis looking at clustered 
use of tools, he identified that a stable state of LMS adoption at his institution had been reached. 
While this was an interesting approach, it was a monumental task to manually examine usage 
across courses and too time consuming to apply and replicate at scale. 

Conceptual studies have since posited models for a more holistic view of institution-wide 
LMS adoption, specifically examining analytics captured by the LMS and their potential use for 
institutional decision-making and instructional intervention. For example, Janossy and Hover 
(2008) proposed a 14-step model for analyzing user behavior within the system that could be 
applied to any LMS through content analysis of system database tables. Dawson, McWilliam, and 
Tan (2008) demonstrated how data captured from the LMS could be used to inform institutional 
decision-making processes and identify potential "at-risk" students. While presenting examples of 
data derived from an institution-wide LMS, they underscored the challenge of readily and 
accurately interpreting data and translating findings to practice. Whitmer (2012) explored the 
interplay of student LMS usage, persistence, and course achievement within a large hybrid-format 
course and found that data from the LMS could be used as a meaningful indicator of student effort. 
The rapidly expanding field of learning analytics continues to push the boundaries of what is 
known, and what can potentially be learned, from usage data stored by the LMS and other 
infonnation technology applications. 

Despite the conceptual interest, few studies have empirically analyzed LMS usage data at 
scale and over time. Fritz (2016) sought through an institution-wide analysis of FMS usage data 
to demonstrate how analytics focused on student use of the FMS could identify effective faculty 
LMS course designs, noting the potential for LMS usage to serve as a proxy for online teaching 
and learning engagement. Park and Jo (2016) analyzed system log data from 7,940 LMS course 
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sites to evaluate learning activity, noting significant gaps in LMS usage levels across colleges and 
disciplines within the institution. More recently, research exploring student use and time on task 
within the LMS has provided an additional perspective to faculty use. Taking tool use as a proxy 
for course design, Whitmer, Nunez, Harfield, and Forteza (2016) sought to discover archetypes of 
course design across institutions through examination of LMS usage. Analyzing an anonymized 
data sample that included 70,000 Blackboard courses from 927 institutions with 3,374,462 unique 
learners, they aggregated tool usage data at the course level based on the percentage of time 
students spent using each tool. As a result, Whitmer et al. identified five course patterns which 
generalized broad approaches to LMS usage which ranged from supplemental use to fully 
integrated holistic use. 

Understanding the relevant factors affecting the adoption and use of the LMS can aid the 
design, development, and implementation of more effective support and training for faculty and 
learners (Kultur & Yazici, 2014). Given the vast and ever-expanding volume of data being 
generated and recorded regarding the use of instructional technology systems, there is a need for 
simple and automated methods for gathering and examining actual usage data. By understanding 
usage patterns of instructional technology tools such as the LMS by faculty members and students, 
institutional support personnel and administration can make better, data-informed decisions 
regarding future technology procurement and support prioritization to help ensure that 
instructional needs are being met. 


Context for Study 

The faculty at one Midwest, public, research-intensive university became interested in 
blended and online teaching in the late 1990s. This grassroot interest was supported through 
training and resources provided by the university's faculty development center. During that time, 
a committee of university faculty explored several LMS options and ultimately chose Blackboard 
(now known as Blackboard Learn). In 2001, a limited pilot was conducted, and Blackboard was 
adopted as an enterprise-wide system in 2002. 

The university's faculty development center was charged at that time with providing 
support for faculty on all aspects of teaching with technology, including teaching using the LMS. 
Due to the university's culture of departmental autonomy, use of the LMS was not mandated and 
training was not required. Blackboard adoption and training have always been voluntary, yet usage 
has grown year after year. Today, nearly all faculty use the system. However, the term "use" can 
be defined in many ways and encompasses a variety of scenarios. 

As adoption has become more saturated and more courses have been offered online, 
training and support have become more sophisticated and differentiated. This added complexity 
strained the capacity of the faculty development center to continue to offer high-quality support. 
Better data and evidence were necessary to prioritize the best level and types of support moving 
forward. 


Research Questions 

Analysis of the LMS adoption and tool usage has developed at the university since 
Blackboard was selected. Initially, the university only tracked the overall usage of the LMS, such 
as the number and percentage of faculty and students who used the LMS for at least one course. 
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As overall adoption increased, it became important to take a more nuanced approach to address 
individual tool usage as well as overall adoption. This study addressed three primary research 
questions: 

1. What LMS tools do faculty include in their courses most often? 

2. How has LMS tool use changed over time? 

3. Does LMS tool use differ based on course modality (face-to-face versus online)? 

In particular, this study considered faculty course design in the use of individual tools within the 
LMS. 


Methods 

This study utilized automated methods to accurately identify how LMS tools were 
implemented across courses and used over time. Previous studies’ manual methods for collecting 
this kind of data were not feasible due to the time and effort involved. Fortunately, data for 
Blackboard Learn is stored in a database and can be queried using custom-written scripts for either 
Oracle or Structured Query Language (SQL) servers. Because the university operates Blackboard 
Learn in a self-hosted environment (which means that the university owns and maintains the 
servers on which the LMS is installed), university staff also have direct access to the main database 
where the data are stored. When universities choose to have Blackboard host the system for them, 
there are still some data which can be extracted from the Open Database. Other LMSs have 
alternate methods for gathering similar system data, as well. 

The SQL queries used in this study were adapted from the work of Kodai (2013). 
University staff customized the scripts to stay current with changes in the structure of the data due 
to upgrades to the LMS, as well as to track additional tools and features unique to the university. 
The SQL script revolved around the COURSECONTENTS table of the database, which tracked 
the tools and content added to courses. Each row of that table represented the use of a tool in a 
given course. Tools were identified by "handles" (CNTHNDLR_HANDLE) and courses were 
identified by unique IDs (CRSMAIN PKl). From there, the script aggregated the data by course 
ID and tool name, and counted the number of records returned for each course-tool pair. 

Occasionally, another table needed to be queried when a tool count could not be aggregated 
in this manner. For example, the Announcement tool was included in each course only once. 
Counting the single instance of the tool did not indicate whether or how much the Announcement 
tool was used. In this case, the query aggregated usage counts based on the ANNOUNCEMENTS 
table, which was where the content of any Announcements were stored. Because these additional 
tables also referenced the same unique identifiers for each course, the query was able to combine 
this with the results from the primary COURSE CONTENTS table. 

University staff ran the SQL script initially against all of the course data which existed in 
Blackboard (back to the spring 2011 semester), and again at the end of every semester. The results 
were compiled in a spreadsheet. Once tool use counts were successfully collected from the 
database, blank records were removed so that the statistics were not reduced by unused Blackboard 
course sections. Only courses with tool counts greater than zero were kept in the dataset. This 
included a few courses which did not use the LMS, but ensured all courses with minimal tool use 
were included. This process recognized that there was a wide range of legitimate use of Blackboard 
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for teaching, and that it was not always possible to distinguish courses taught using Blackboard 
from those minimally used for other purposes. 

Finally, the counts of tool usage were converted using a dummy coding method that only 
indicated whether or not a tool was used. For every course where a tool was used, the tool was 
coded as a 1, and coded as a 0 when the tool was not used. This binary approach was used as an 
initial analysis tool, to determine whether a tool was utilized within a course. There are interesting 
pedagogical and technology-integration implications for the number of times a tool was used (such 
as whether a course included 1 grade per student or 30 grades per student, or if 5 announcements 
were posted or 50), but such investigation is reserved for a future study. For the current research, 
the dummy coded values were used to compute the percentage of courses that used each tool in a 
given semester. 

A significant aspect of using this approach was data verification and error-checking, at 
multiple steps in the process. Because the Blackboard LMS has evolved over time, upgrades and 
patches occasionally necessitated changes to the script. For example, the built-in plagiarism 
detection tool was originally a stand-alone feature, but then became an option enabled on the 
primary Assignment tool. This also included a change in the data-structure, which was detected 
during routine verification measures. Another example is the addition or replacement of tools that 
required new or updated scripts, such as the migration from Wimba Classroom to Blackboard 
Collaborate, and more recently to Blackboard Collaborate Ultra. Each of these changes required 
significant changes and extensive validation. 

When the queries were run, the process was monitored to ensure that the operations 
completed successfully. In the event that one or more scripts failed, the queries were revised and 
retried. If the scripts ran successfully, the results were scanned visually for any anomalies, such as 
tools reporting no usage in any course. Finally, the tool usage percentages and trends were 
inspected for data consistency, including unusual spikes or steep changes. In difficult cases where 
the queries were particularly challenging to refine, staff built test courses for comparison against 
query results to identify the appropriate table entry for gathering usage data. To ensure adequate 
understanding of the results, staff also developed a detailed data dictionary that defined how each 
tool was measured. 


Results 

The results of the analysis show that the use of Blackboard for credit-bearing courses has 
grown substantially over the years at the university. Use of Blackboard by faculty rose from 65.5% 
of all instructional staff in the fall of 2008, which includes faculty, instructors, and teaching 
assistants who are instructor of record for a course, to a peak of 92.1% in fall of 2015, before 
decreasing slightly to 87.9% in fall of 2016 (Figure 1). Over the same time period, student use has 
remained consistently high, in the low to mid-90%. Since student use is both driven and limited 
by faculty use of Blackboard, the high percentage of students using it while faculty use was fairly 
low speaks to the widespread usage across campus. 
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2008 2009 2010 2011 2012 2013 2014 2015 2016 

Year 


Figure 1. Percent of students and instructional staff who used Blackboard in the fall semester of 
each year. 


In addition, further analysis of the courses that used Blackboard showed that only 41.9% 
of all course sections used Blackboard during the fall 2010 semester (Figure 2). However, over 
time that has increased, and 63.8% of all courses sections used Blackboard during the fall 2016 
semester. Interestingly, as with overall instructional staff use, the percentage of courses using 
Blackboard also peaked in the fall semester of 2015, at 67.0%. For this purpose, “use” is defined 
as any course section requested by faculty in Blackboard. The university requires faculty to submit 
an automated request for their Blackboard course to be created. This process also synchronizes the 
course with the student information system, so that the Blackboard course can be automatically 
populated with the students who were enrolled in the course. This definition of use indicates that 
faculty intended to use the Blackboard course, but it does not consider whether faculty did in fact 
make use of the course, or how they did so. In addition, although it is clear that the majority of 
faculty were using Blackboard, and adoption rate is an important indicator of the success of a 
technology implementation, it does not sufficiently answer questions regarding how the 
technology was used, and how its use had changed over time. 
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Semester 


Figure 2. Percent of course sections, which used Blackboard in the spring, summer, and fall 
semesters of each year. 


The most frequently used tools in the fall 2016 semester were Announcements, Items, 
Grades, Folders, Files, Assignments, Web Links, Plagiarism Detection, Discussion Boards, and 
Tests, in that order. Descriptions of each tool and the usage in the fall 2016 semester are in Table 
1. While these tools are specifically available in Blackboard Leam, they are common elements of 
most LMSs. After selecting tools based on prevalence in the fall 2016 semester, the same analysis 
technique was used to track the usage of those tools for every tenn in which data were available, 
back to the spring semester of 2011 (Figure 3). For individual tools, “use” was defined as the tool 
being deployed or activated at least once within a course. The percentages were calculated as the 
number of courses in which the tool was used compared with the number of courses in which at 
least one tool was used (excluding courses which may have been requested, but were not used). 
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Tool _ 

Announcements 

Items 

Grades 

Folders 

Files 

Assignments 

Web Links 

Plagiarism Detection 

Discussion Boards 

Tests 


Description _ 

timely reminders posted within the LMS 
and optionally sent as email 

content type that includes either one or 
more files or fonnatted text created using a 
WYSIWYG editor, or both 

one or multiple columns of grades recorded 
in the Grade Center, for the purposes of 
tracking scores and calculating a total grade 
and/or communicating scores and feedback 
to students 

content type used to provide organizational 
structure for other content and assessments 

content type that allows the upload/posting 
of a single file, without any accompanying 
text 

assessment that allows students to submit a 
previously created file for grading, 
facilitates in-browser grading and integrates 
with Grade Center 

content type used to provide a hyperlink to 
a website, with optional attached files or 
fonnatted text 

a setting available for Assignments to 
compare the content of student-submitted 
files to other submissions and web-based 
content to identify potentially-plagiarized 
work 

threaded discussion with at least one reply 
posted 

an assessment that includes one or more 
questions of a variety of both objective or 
subjective question types (e.g., multiple 
choice, fill in the blank, matching, short 
answer, essay) 


Percent of 
courses using in 
Fall 2016 
semester 
82.13% 

76.79% 

70.61% 

62.30% 

53.33% 

52.60% 

29.88% 

22.34% 

21 . 22 % 

19.30% 


Table 1. Ten Most Commonly Used Tools in the Fall 2016 Semester 
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— ♦ - Announcements 
Items 

—*—Grades 
-■X--Folders 
-X- Files 
—O— Assignments 
—»—Web Links 

Plagiarism Detection 
■ Discussion Boards 
• Tests 


Figure 3. Percent of all Blackboard courses using each of the ten most-used tools. 


The analysis shows that use of most tools generally increased over the approximately five- 
year time period. While use of some tools increased more than others, all of the top ten tools were 
used in a higher percentage of courses in the fall 2016 semester than in the 2011 spring semester. 
There was a marked decrease in the use of Items from spring 2011 to fall 2011, while Files rose. 
That May, after the end of the spring 2011 semester, the university upgraded from Blackboard 
Learn 8 to Blackboard Leam 9.1, which introduced Files as a new content type. The simultaneous 
decrease in use of Items and increase in use of Files is evidence that some faculty adopted Files 
and stopped using Items. As Files became more prevalent, Items again increased in usage, 
demonstrating side-by-side use of the two tools. 

There was a steeper increase in use of several tools in the fall semester of 2015, including 
Announcements, Items, Grades, Files, and Assignments. This was likely due to an ongoing 
campaign to reduce printing by both faculty and students. Students were responsible for paying 
for their printing as of the fall 2015 semester (Ervins, 2012), which would have encouraged faculty 
to use digital assessments more. As a result, this would have led to increased use of Grades as well, 
because the assessment tools feed into the Grade Center. 

The visualization of the usage trends shows interesting peaked behavior for most tools 
during each summer semester, as a higher percentage of courses used tools during those semesters. 
This is particularly noticeable for Folders, Assignments, Discussion Boards, and Tests. There may 
be several contributing factors that led to this behavior. The primary reason for an increase in tool 
use in the summer seems to be increased online course offerings each summer. Figure 4 shows 
that online courses represented a much higher percentage of overall course offerings in the summer 
semester than in either fall or spring semester. In Figure 4, the online courses category includes 
those offered both fully online or online with face-to-face meetings, whereas the face-to-face 
courses category includes traditional face-to-face courses as well as independent study, internship, 
dissertation/thesis, and student teaching courses. This behavior closely mimics the peaks seen in 
the tool usage, so the next step was to examine tool usage based on the course modality. 
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-•-Face-to-Face Courses 
—♦—Online Courses 


Figure 4. Percent of courses offered face-to-face or online for each semester. 


Interestingly, face-to-face courses showed some of the same peaked behavior for tool use 
seen in the initial combined data. Figure 5 shows that Files, Grades, and Assignments were used 
at somewhat higher rates during each summer semester in face-to-face courses than they were used 
during the fall and spring semesters. Because most courses have a compressed schedule when 
offered during the summer semester, at only 8 weeks long as opposed to 16 weeks long during fall 
and spring semesters, faculty may have relied on more Blackboard tools to support instruction or 
to reduce in-class time by blending instructional activities into the online environment. Overall, 
though, tool usage showed less variation and volatility in face-to-face courses than in the combined 
data. 



Announcements 

Items 

—*—Grades 
-■X--Folders 
-x- Files 
—O— Assignments 
—Web Links 

Plagiarism Detection 
■ Discussion Boards 
♦ Tests 


Figure 5. Percent of face-to-face courses using Blackboard which use each of the ten most-used tools. 
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By comparison, online courses used nearly all of the LMS tools at higher rates than face- 
to-face courses (Figure 6). For example, eight of the top ten tools were used in at least 50% of 
online courses in the fall 2016 semester. In addition, six of the tools were used in over 75% of 
online courses in fall 2016 (Items, Announcements, Folders, Assignments, Grades, and Discussion 
Boards, in descending order of prevalence). This is to be expected, since online courses rely 
heavily on technology to support student learning. 



♦ Announcements 
Items 

—*—Grades 
--X-- Folders 

— X- Files 

—O— Assignments 
—+—Web Links 

Plagiarism Detection 

- « ■ Discussion Boards 

♦ Tests 


Figure 6. Percent of online courses using Blackboard which use each of the ten most-used tools. 

The two tools which were used less frequently in online courses compared with face-to- 
face courses were plagiarism detection and files. Plagiarism detection was used at a rate consistent 
with face-to-face courses. In the fall 2016 semester, 23.55% of online courses and 22.26% of face- 
to-face courses used plagiarism detection. Usage was fairly stable from summer 2015 through fall 
2016 for online courses, while face-to-face courses saw a substantial drop in usage in summer 
2016. Files were used slightly less often in online courses in the fall 2016 semester with 49.17% 
of online courses and 53.76% of face-to-face courses posting content as Files. Given that Items 
were one of the most used tools for online courses, the lower usage of Files does not represent less 
content being posted in online courses, but rather a stronger preference for Items as the means for 
posting content in the LMS. This may be because the Item tool allows the faculty to provide 
additional context or instructions using the text editor, which is unavailable for Files. Further 
investigation is needed to determine the motivation for using one tool over the other. 


Discussion 

Initially, this study was undertaken to improve LMS support within the university. 
Understanding how the LMS is currently used has helped to inform the workshops and resources 
offered by the university faculty development center. For example, the center can prioritize topics 
for development of online tutorials based on the most-used tools, such as advanced techniques for 
grading Assignments using integrated rubrics. The results have also influenced the center's 
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summer staffing models for LMS support. In the past, it was assumed that support requests would 
be lower in the summer because there are fewer courses offered. However, the increased usage 
rate of LMS tools in the summers challenged that assumption, forcing the center to rethink its 
strategy, and ensure adequate staff coverage for the summer months. 

The data also helped to visualize a bigger picture of the use of the system. In general, a 
support unit interacts with a fairly biased sample of faculty, namely those that either struggle with 
the system and therefore experience problems or those who excel at using the system and use it in 
advanced ways, and thus experience problems. While data on support staff interactions can be used 
to inform the practices of a faculty development center (Krishnamurthi & Rhode, 2013), it 
provided a limited view due to this sample bias. Considering the actual use of the system provided 
more scope to inform decision-making. There were other sources of data to consider, as well, such 
as website analytics, that could provide insight into the demand or gaps of existing resources 
(Rhode, Richter, Gowen, & Krishnamurthi, 2015). 

There are number of limitations to such a study, however. First, using do-it-yourself data 
collection using database queries required significant amounts of verification to demonstrate that 
the queries were gathering accurate results. Because of this, data gathering may be iterative and 
incremental as opposed to comprehensive, as the process was refined. In this study, there were a 
few LMS tools about which data could not be gathered, such as web conferencing sessions, 
because such data was not available to a database query. For other tools, usage data was held in 
another system, making it difficult or impossible to aggregate with the course-based tool usage 
data. Web conferencing was again an example of this; the institution's primary web conferencing 
system stored usage data in a separate server hosted by the provider, and records session and user 
data but not course, making it impossible to report web conferencing usage in the same fonnat as 
the other LMS tools. As more technologies external to the LMS are used and integrated with the 
LMS, such as via a learning tools interoperability (LTI) standard or with publisher provided 
systems, data gathering will continue to become more difficult. This approach was best-suited for 
tools native to the LMS. 

Another limitation to the data is that some information is simply not available to the 
researchers. For example, specialized courses, including independent study, internship, 
dissertation/thesis, and student teaching cannot also be designated as online in the university's 
student information system. It is possible that some sections of these specialized courses are 
actually considered to be online courses (such as when the section is dedicated for students in an 
online degree program), but that infonnation is not currently recorded. Also, the researchers did 
not have access to other data sets that would be obvious to include in research on LMS usage, such 
as course grades, achievement of student learning outcomes, or faculty and course evaluations. It 
would be worthwhile to investigate whether specific tools or combinations of tools impact these 
types of outcomes, but requires data management structures or protocols not currently in place at 
the institution. 
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Conclusion 

This is just the beginning of using LMS tool usage data to investigate patterns of teaching 
and learning. At this one institution, LMS adoption overall may not be increasing because use has 
reached the saturation point. Total (100%) adoption is not likely to occur, and is not even a 
desirable outcome. However, it is clear that individual tool use is still increasing, with obvious 
differences between online and face-to-face course modalities. 

There are several future directions that study of LMS usage should consider. For example, 
this study only considered descriptive statistics of usage trends, and further statistical analysis is 
needed. In addition to considering individual tools, there is much to be learned about clusters or 
combinations of tools. Which tools are used together and create synergies of practice? In addition 
to the differences by course modality, are there also differences to be seen by academic discipline 
or by pedagogical approach? While using LMS database logs provides a comprehensive and 
unbiased view of the use of LMS tools, it cannot interpret intention or motivation for use. Future 
research should also consider why faculty use the LMS in general, or what their pedagogical 
intention is for adopting individual or clusters of tools within the LMS. 

Despite the growing popularity of Next Generation Digital Learning Environments 
(NGDLE), which are loosely coupled collections of tools (Brown, Dehoney, & Millichap, 2015a), 
studies such as this show that there is still a need for formal learning management systems to 
provide structure for the learning process. While the NGDLE is an attractive application of the 
more advanced cloud-based tools and emerging protocols such as LTI and xAPI, the extensive use 
of the LMS and of the variety of tools used in each course indicate that the more flexible NGDLE 
environment may also require too much time and knowledge for individual faculty to implement. 
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